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FOREWORD 


The  Intelligent  Training  Systems  Technologies  Team  of  the 
Fort  Gordon  Field  Unit  of  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  performs  research  on  and 
develops  new  and  innovative  training  technology.  This  work  is  an 
essential  part  of  ARI's  mission,  which  is  to  conduct  research  to 
improve  the  Army's  capability  to  effectively  and  efficiently 
train  personnel. 

The  job  performance  aid  described  in  this  report  was  devel¬ 
oped  to  fulfill  a  military  need  for  assistance  in  training  and 
sustaining  skills  for  operating  the  Mobile  Subscriber  Radio- 
Telephone  Terminal  (MSRT) .  The  job  aid  was  developed  at  the 
request  of  Brigadier  General  Alfred  J.  Mallette,  Deputy  Command¬ 
ing  General,  USASC&FG. 

This  research  was  guided  by  a  Memorandum  of  Agreement  be¬ 
tween  ARI  and  the  United  States  Army  Signal  School,  Fort  Gordon, 
for  the  establishment  of  an  ARI  Training  Research  Element  at  Fort 
Gordon,  Georgia,  dated  1987.  The  research  was  conducted  by  the 
Intelligent  Training  Systems  Technologies  Team  at  ARI  Fort  Gordon 
Field  Unit  under  the  ARI  program  task  entitled  "Acquisition  and 
Retention  of  Communication  and  Electronics  Skills." 

The  development  and  evaluation  efforts  were  reviewed  and 
approved  by  the  Training  and  Doctrine  Command  (TRADOC)  System 
Manager  for  Mobile  Subscriber  Equipment  (TSM-MSE)  Unit  at  Fort 
Gordon.  The  TSM-MSE  will  distribute  the  job  aid  to  non-signal 
MSRT  users.  The  job  aid  effort  was  also  presented  in  a  poster 
session  to  the  Military  Psychology  Division  of  the  98th  Annual 
Convention  of  the  American  Psychological  Association,  Boston, 
Massachusetts,  August,  1990. 


EDG/W  M.  JOHNSON 
Technical  Director 
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THE  DEVELOPMENT  AND  EVALUATION  OF  A  JOB  AID  TO  SUPPORT  MOBILE 
SUBSCRIBER  RADIO-TELEPHONE  TERMINAL 


EXECUTIVE  SUMMARY 


Requirement: 

Operators  of  the  Mobile  Subscriber  Radio-Telephone  Terminal 
(MSRT)  require  support  to  learn  and  retain  the  skills  necessary 
to  operate  the  equipment.  Because  the  MSRT,  a  piece  of  Signal 
equipment,  will  be  fielded  to  non-Signal  soldiers,  the  need  for 
training  support  is  greater  than  it  would  be  for  other  equally 
complex  and  critical  tasks.  These  non-Signal  soldiers  will  re¬ 
ceive  limited  formal  training  in  operation  of  the  MSRT,  and  the 
amount  and  type  of  informal  training  received  will  vary  from 
company  to  company.  To  offer  standard  assistance  in  this  task, 
the  MSRT  job  aid  was  developed  for  the  non-Signal  soldier. 


Procedure: 

Twenty-nine  non-Signal  soldiers  from  Fort  Gordon  attempted 
to  operate  the  MSRT  under  three  experimental  conditions.  One 
group  operated  the  MSRT  using  the  ARI-developed  job  aid  only  (JA 
ONLY  group) .  A  second  group  used  the  job  aid  and  received  a 
short  demonstration  of  the  operating  procedures  (JA  +  DEMO 
group) .  A  third  group  operated  the  MSRT  using  the  Army  Technical 
Manual  (TM)  for  the  task  and  received  the  same  short  demonstra¬ 
tion  (TM  +  DEMO) .  Before  operation  of  the  MSRT,  subjects  were 
given  the  Test  of  Adult  Basic  Education  (TABE)  Vocabulary  and 
Comprehension  subtests  as  covariate  measures.  Rank  and  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  Electronic  (EL)  and 
Surveillance  Communication  (SC)  scores  were  also  used  as  cc  'ari- 
ates.  Procedure  completion  time,  fatal  errors,  and  nonfatal  er¬ 
rors  were  collected  on  each  subject  during  operation.  P.»^sponses 
to  a  satisfaction/confidence  survey  were  collected  frc  each  sub¬ 
ject  after  operation. 


Findings: 

Significant  results  were  found  between  all  groups  on  the 
time  to  complete  variable.  The  JA  +  DEMO  group  performed  the 
procedures  faster  than  the  other  two  groups  and  the  JA  ONLY  group 
performed  faster  than  the  TM  +  DEMO  group  Significant  results 
were  also  found  on  the  satisfaction/confidence  survey:  both  JA 
groups  claimed  more  overall  satisfaction  and  confidence  with  the 
task  than  the  TM  +  DEMO  group.  Significant  results  were  not 
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found  across  the  two  error  variables;  this  was  explained  by  the 
power  analysis. 


Utilization  of  Findings: 

The  job  aid  has  been  transferred  to  the  Fort  Gordon  Training 
and  Doctrine  Command  System  Manager  for  Mobile  Subscriber  Equip¬ 
ment  for  approval  and  distribution.  It  is  being  distributed  to 
MSRT  users.  It  is  the  expectation  that  the  job  aid  will  be  used 
to  improve  non-Signal  soldiers'  ability  to  communicate  effi¬ 
ciently  in  tactical  environments. 
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THE  DEVELOPMENT  AND  EVALUATION  OF  A  JOB  AID  TO  SUPPORT 
MOBILE  SUBSCRIBER  RADIO-TELEPHONE  TERMINAL 


Introduction 

In  an  effort  to  keep  pace  with  evolving  technologies  and  a 
limited  economy,  the  U.S.  Army  has  initiated  programs  to  streamline 
training  procedures.  The  end  result  of  this  effort  will  be  more 
cost-effective  training  methods  and  practices.  One  area  of 
recent  change  in  both  technology  and  training  requirements  is  the 
addition  of  Mobile  Subscriber  Equipment  (MSE)  to  the  repertoire 
of  the  Signal  soldier.  Advances  in  the  development  of  MSE  have 
generated  complex  communications  devices  that  require  the  human 
operator  to  recall  greater  amounts  of  information  than  required 
for  earlier  equipment. 

The  extremely  complex  MSE  system  offers  secure  voice,  data, 
and  facsimile  communications  to  soldiers  at  the  corps  and 
division  levels  throughout  the  tactical  area.  Soldiers 
communicate  within  the  MSE  network  primarily  through  the  Mobile 
Subscriber  Radio-Telephone  Terminal  (MSRT) ,  which  consists  of  the 
Digital  Subscriber  Voice  Terminal  (DSVT)  and  the  RT  1539  radio. 

When  the  MSE  system  has  been  fully  fielded,  over  13,000  MSRTs 
will  be  in  use  throughout  the  Army.  Approximately  9,500  of  these 
will  be  used  by  soldiers  occupying  non-Signal  Military 
Occupational  Specialties  (MOS) . 

The  complexity  and  criticality  of  efficient  MSRT  operation, 
as  well  as  the  problem  of  training  non-Signal  users  on  a  Signal 
task,  has  provided  the  impetus  for  the  U.S.  Army  Research  Institute 
(ARI)  to  analyze  the  need  for  a  job  performance  aid  for  this 
task.  A  job  aid  has  been  defined  as  a  physical  memory  aid  that 
provides  information  that  guides  or  facilitates  on-the-job 
performance  (Winn  &  Evensen,  1988) .  Several  types  of  job 
performance  aids  are  in  common  use  in  the  U.S.  Army;  they  include 
check  lists,  work  sheets,  and  note  pads  (Evensen,  Winn,  and 
Salter,  1988) .  Such  job  aids  have  been  developed  for  use  by 
Combat  Leaders  (Winn  &  Evensen,  1988) ,  and  M60A3  tank  crews 
(Morrison,  1985;  Kraemer,  Anderson,  Kristiansen,  &  Jobe,  1985). 

Based  on  the  User's  Manual  for  Predicting  Military  Task 
Retention,  (Rose,  Radtke,  Shettel,  &  Hagman,  1985) ,  30  percent  of 
Signal  soldiers  attempting  to  operate  the  MSRT  after  a  two  month 
period  of  nonuse  can  be  expected  to  fail.  Factors  that 
contribute  to  this  prediction,  and  therefore  contribute  to  the 
difficulty  of  skill  maintenance  for  MSRT  operation,  include;  task 
length  and  difficulty,  skill  requirements,  and  the  quality  of  the 
job  aid,  if  any,  for  the  task.  The  model  predicts  that  high 
quality  job  aids  can  reduce  the  rate  of  skill  decay. 
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This  skill  decay  prediction  indicates  a  serious  need  for  some 
type  of  assistance  for  soldiers  performing  MSRT  tasks.  This 
requirement  is  compounded  by  the  fact  that  the  MSRT  will  be 
operated  by  non-Signal  soldiers  while  the  predictions  were  based 
on  the  assumption  that  the  equipment  would  be  operated  by  well- 
trained  Signal  soldiers. 

Many  guidelines  exist  to  help  determine  whether  a  job  aid  is 
appropriate  (Lenzychi  and  Finley,  1980) .  Written  job  aids  are 
advocated  for  use  with  tasks  that  are  so  critical  and  complex 
that  they  should  not  be  attempted  from  memory  alone,  such  as  MSRT 
operation.  The  magnitude  of  the  impact  of  criticality  and 
complexity  on  performance  increases  exponentially  when  performed 
by  equipment-naive  non-Signal  soldiers.  Furthermore,  non-Signal 
soldiers  may  not  need  to  use  MSRT  equipment  for  long  periods  of 
time,  and  job  aids  are  also  advocated  when  information  must  be 
retained  during  periods  of  long  disuse. 

Among  the  most  difficult  types  of  procedures  to  recall  in 
proper  sequence  are  those  that  are  not  cued  by  prior  steps  or  by 
equipment  indications  (Farr,  1986) .  The  MSRT  procedures  require 
many  actions  that  are  not  clearly  cued.  Although  retention  is 
improved  by  practice  or  continued  training,  the  trainee  may 
become  dependent  on  specific  contextual  cues  causing  performance 
to  suffer  (Lane,  1986) .  One  role  of  the  written  job  aid  is  to 
shorten  training  programs  by  providing  specific  retrieval  cues  on 
paper,  so  that  overlearning  is  not  necessary.  Thus,  an  effective 
job  aid  for  MSRT  operation  could  reduce  training  time  and  costs, 
while  enabling  soldiers  to  maintain  a  suitable  level  of 
performance. 

In  order  to  ensure  that  the  infrequent  non-Signal  user  of  the 
MSRT  can  maintain  full  operational  effectiveness,  in  the  face  of 
all  the  presented  problems,  ARI  has  developed  a  job  aid  for 
equipment  operation.  The  development  of  written  job  aids  begins 
with  the  process  of  task  analysis  (Foley,  Joyce,  Mallory  & 

Thomas,  1971) .  Task  level  data  is  converted  to  the  proper  job 
performance  aid  format.  The  TM  for  MSRT  operation  (Technical 
Manual  11-5820-1021-10,  1989)  was  studied  carefully  and  a 
checklist  was  formulated  and  converted  to  a  military  format 
suggested  by  the  Signal  Leadership  Department  (SLD) ,  Ft.  Gordon. 
Subject  Matter  Experts  (SMEs)  were  consulted  extensively  in 
troubleshooting  the  job  aid  content  and  format. 

The  content  of  the  checklist  was  intended  to  cover  all 
information  necessary  to  operate  the  MSRT  under  normal 
circumstances.  That  is,  the  job  aid  was  intended  to  be 
sufficient  for  use  by  an  untrained  soldier.  The  job  aid  was 
piloted  several  times  to  ensure  that  it  was  readable  and  simple 
to  follow.  SMEs  also  helped  identify  procedures  to  include  in 
the  job  aid.  Six  critical  operating  procedures  are  represented: 
Power  Up,  RT1539  Cryptovariable  Load,  DSVT  Cryptovariable  Load, 
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Manual  Frequency  Plan  Load,  Affiliation  and  Make  a  Call.  (See 
Appendix  A  for  a  copy  of  the  job  aid.)  This  list  does  not 
include  all  possible  procedures  for  MSRT  operation,  but  rather  is 
intended  to  represent  the  critical  tasks.  A  detailed  description 
of  the  task  can  be  found  in  the  Technical  Manual  (TM  11-5820- 
1021-10,  1989). 


Method 


Subjects 

Twenty  nine  non-Signal  soldiers  stationed  at  Fort  Gordon 
participated  in  the  evaluation.  The  rank  of  the  subjects  ranged 
from  PVT  to  SSG.  Nineteen  were  male  and  ten  were  female. 

Material  and  Equipment 

Paper-based  material  included  the  ASVAB  scores,  TABE  scores, 
(described  under  "covariates) ,  the  questionnaire  developed  by  API 
(described  under  "dependent  variables",  or  see  appendix  B) ,  the 
Army  Technical  manual  and  the  job  aid  itself. 

A  typical  vehicle  mounted  MSRT  was  used  to  collect 
performance  data.  It  was  not  connected  to  the  communications 
network . 

Covariates 


Variables  used  to  covary  within  cell  error  variance  included 
rank,  the  Test  of  Adult  Basic  Education  (TABE)  Vocabulary  and 
Comprehension  subscores,  and  the  ASVAB  EL  and  SC  subscores.  The 
ASVAB  subscores  were  obtained  from  the  Army  and  reflect  scores  on 
aptitude  tests  administered  when  the  soldiers  entered  the  Army. 
These  scores  were  five  to  ten  years  old. 

Dependent  Variables 

Three  dependent  measures  were  collected  to  quantify  the 
performance  of  the  subjects  on  each  of  the  six  procedures.  One 
of  the  dependent  variables  was  the  amount  of  time  taken  by  the 
subject  to  complete  each  procedure.  The  other  two  dependent 
variables  estimated  the  number  of  fatal  and  nonfatal  errors 
committed  while  performing  a  procedure.  To  quantify  performance 
on  *he  entire  task,  time  and  errors  were  collapsed  across  the  6 
procedures . 

A  fatal  error  was  defined  as  an  error  that,  if  uncorrected, 
would  prevent  successive  MSRT  operations,  for  example,  failure  to 
turn  the  "Remote  Power  Switch"  on.  Nonfatal  errors  were  defined 
ar  either  an  omission  or  an  incorrect  action  that  did  not 
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adversely  affect  the  rest  of  the  operations.  For  instance, 
forgetting  to  check  the  Loaded  Frequency  Plan  indicator  after  a 
correct  load  was  defined  as  a  nonfatal  error;  it  does  not  halt 
the  operation,  but  is  still  an  important  oversight. 

Finally,  a  satisfaction/confidence  survey  was  devised  to 
assess  the  impact  of  the  different  job  aid  conditions  on  the 
attitudes  and  confidence  of  the  subjects.  Each  question  was 
answered  on  a  5-point  scale.  (See  Appendix  B  for  a  full 
questionnaire) .  Question  l  asked  the  subjects  to  rate  the 
quality  of  the  demonstration  and  was  given  only  to  the  subjects 
in  the  two  demonstration  conditions.  Questions  2  through  4 
required  all  of  the  subjects  to  rate:  the  effectiveness  of  the 
paper-based  aid  that  they  used,  whether  TM  or  the  ARI  job  aid; 
the  difficulty  of  the  task;  and  their  confidence  in  their  ability 
to  perform  MSRT  operating  procedures  in  the  future.  Responses  to 
questions  2  through  4  were  combined  to  estimate  the  subjects' 
overall  satisfaction  with  the  experience. 

Independent  Variables 

In  order  to  evaluate  the  effectiveness  of  the  ARI  developed 
job  aid,  three  experimental  conditions  were  compared.  The 
conditions  differed  in  the  type  of  paper-based  aid  that  was 
offered,  and  whether  the  aid  was  accompanied  by  a  short 
demonstration.  The  three  conditions  were:  ARI  developed  job  aid 
+  demonstration  (Job  Aid  +  Demo),  ARI  developed  job  aid  without 
demonstration  (Job  Aid  Only) ,  and  Army  Technical  Manual  with 
demonstration  (TM  +  Demo) . 

The  decision  was  made,  based  on  pilot  data,  to  include  a  short 
demonstration  with  the  job  aid  in  this  evaluation.  Pilot 
subjects  experienced  difficulty  in  discriminating  between 
components,  and  with  small  manipulation  details  that  proved 
easier  to  demonstrate  quickly  than  to  describe  with  text.  Thus 
the  purpose  of  this  study  is  not  only  to  evaluate  the 
effectiveness  of  the  ARI  develop  3d  job  aid,  but  also  the  effect 
of  adding  a  short  demonstration  to  augment  performance.  This 
addition  partially  addresses  a  question  of  how  much  training 
support  is  required  when  a  job  aid  is  present.  It  is  possible 
that  a  short  demonstration,  to  get  the  subjects  physically 
familiar  with  the  equipment,  is  sufficient  to  smooth  minor 
problems  in  using  the  job  aid. 

The  Job  Aid  Only  group  was  included  in  the  evaluation  to 
assess  the  impact  of  the  demonstration  on  MSRT  performance  by 
comparing  the  performance  of  this  group  to  that  of  the  Job  Aid  + 
Demo  group.  A  finding  of  a  minor  difference  would  indicate  that 
a  demonstration  does  not  amplify  the  effect  of  the  Job  Aid  on 
performance  of  MSRT  procedures. 
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The  two  demonstration  conditions,  i.e.  Job  Aid  +  Demo,  cind  TM 
+  Demo,  were  included  in  the  evaluation  to  compare  the 
effectiveness  of  the  job  aid  to  the  effectiveness  of  a  standard 
military  Technical  Manual.  The  demonstration  was  included  with 
the  TM  to  avoid  unfair  support  in  favor  of  the  job  aid.  This  is 
an  important  comparison  because  the  TM  is  the  only  job  aid 
available  to  operate  most  Army  equipment.  This  comparison 
assumes  that  the  two  demonstrations  were  equivalent. 

The  performance  of  individuals  in  the  TM  +  Demo  condition  and 
Job  Aid  Only  condition  were  compared  to  determine  if  soldiers 
could  more  effecti\'ely  operate  the  equipment  with  only  the  job 
aid.  A  finding  of  a  difference  between  these  two  groups 
eliminates  the  possibility  that  the  Job  Aid  +  Demo  condition 
outperforms  the  TM  +  Demo  condition  because  of  a  differer.  e 
between  the  effectiveness  of  the  two  demonstrations. 

Procedure 


Groups  of  five  or  six  subjects  were  evaluated  per  day  for  five 
days,  for  a  total  of  approximately  30  subjects. 

Orientation.  Each  day  the  subjects  arrived  at  ARI  at  8:00 
a.m.  and  were  given  a  fifteen  minute  orientation,  which  explained 
the  purpose  of  the  evaluation,  and  presented  an  outline  of  the 
day's  activities.  After  the  general  outline,  the  purpose  of 
collecting  the  TABE  subscores  was  explained  and  a  more  specific 
description  of  the  test  was  given.  Then,  the  subjects  were  asked 
to  sign  a  form  indicating  their  understanding  that  all  the  scores 
would  be  kept  confidential. 

TABE  Test.  Next,  the  day's  subject  group  was  escorted  to  a 
learning  center  on  post  to  take  the  TABE  subtests.  Learning 
center  staff  administered  and  scored  the  tests.  After  the 
subjects  were  given  the  maximum  time  to  take  these  subtests,  40 
minutes,  they  were  given  another  orientation,  which  explained  in 
more  detail  the  activities  for  the  rest  of  the  day.  Subjects 
were  scheduled  at  45  minute  intervals  to  return  to  ARI  to  operate 
the  MSRT. 

MSRT  Operation.  At  the  appointed  time,  each  subject  received 
the  paper-based  aid  (TM  or  ARI  job  aid)  for  the  assignea 
condition  to  study  for  15  minutes.  The  subject  was  told  to  use 
the  time  to  familiarize  themselves  with  the  organization  of  the 
material,  rather  than  attempting  to  memorize  procedures.  At  the 
end  of  the  15  minute  study  period,  the  subject  was  taken  outside 
and  asked  to  operate  a  vehicle-mounted  MSRT. 

Before  MSRT  operation  began,  subjects  in  both  demo  conditions 
were  given  a  short  (average  time-4  minutes)  demonstration 
consisting  of  an  introduction  to  the  major  components  of  the 
MSRT,  and  a  step-by-step  operation  of  the  equipment.  Also, 
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subjects  in  the  TM  +  Demo  group  were  told  that  some  actions 
listed  in  the  TM  were  not  necessary  and  that  the  experimenters 
vould  give  instructions  when  a  step  should  be  skipped.  All 
subjects  were  notified  that  the  equipment  would  not  be  fully 
operational  because  the  MSRT  was  not  connected  to  the  MSE 
network,  and  that  the  experimenters  would  intercede  if  the 
disconnect  status  interfered  with  MSRT  operations. 

After  all  the  information  was  imparted,  the  subject  was  asked 
to  attempt  to  operate  the  MSRT,  one  procedure  at  a  time.  The 
subject  was  instructed  to  begin  when  asked  to  do  so,  but  to  stop 
when  finished.  However,  any  subject  who  did  not  finish  a 
procedure  in  ten  minutes  was  stopped  by  the  experimenters.  Two 
experimenters  recorded  the  time  and  the  number  of  errors  for  each 
procedure  and  the  two  sets  of  values  were  averaged  to  better 
estimate  these  performance  values. 

Satisfaction/Confidence  Questionnaire.  Finally,  after 
operating  the  MSRT,  each  subject  was  asked  to  complete  the 
satisfaction/confidence  survey,  thanked  for  their  time,  and 
dismissed. 


Results 


Descriptive  Statistics 

Summary  statistics  for  the  dependent  measures  and  covariates 
are  contained  in  Table  1.  The  data  were  analyzed  by  the 
Statistical  Package  for  the  Social  Sciences  (SPSS)  program, 
MANOVA.  The  independent  variable  was  the  three-level  grouping 
variable.  The  ASVAB  EL  and  SC  scores,  the  TABE  Comprehension 
and  Vocabulary  scores,  and  rank  were  entered  as  covariates  in  the 
MANOVA  design. 
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Table  1 


Group  Means  and  Standard  Deviations  across  Variables 


Variable 

JA+Demo 

Mean 

1 

SD 

JA  Only 
Mean 

SD 

TM+Demo 

Mean  SD 

Time 

10.22 

5.47 

15.75 

4.50 

23.04 

5.27 

Combined  Survey 

13.60 

1.96 

13.56 

1.33 

9.50 

3.78 

Individual  Survey  Questions: 

1.  Rate  Demo  4.60  .52 

4.00 

.87 

2.  Rate  TM/JA 

4.80 

.42 

4.56 

.53 

3.50 

1.43 

3 .  Rate  Task 

4.30 

1.06 

4.56 

.73 

2.90 

1.10 

4.  Confidence 

4.50 

.71 

4.44 

.53 

3.10 

1.37 

Fatal  errors 

3.20 

3.43 

3.33 

2.00 

5.90 

5.00 

Nonfatal  errors 

3.50 

1.58 

2.00 

2.24 

4.10 

2.69 

Army  Rank 

4.70 

1.34 

4.89 

1.05 

5.00 

.82 

TABE  Voc 

11.77 

1.83 

12.28 

1.25 

11.71 

2.15 

TABE  Comp 

10.62 

2.16 

11.36 

1.14 

11.16 

2.07 

ASVAB  EL 

93.75 

6.60 

93.50 

13.54 

99.88 

14.71 

ASVAB  SC 

91.25 

10.44 

93.25 

11.56 

100.87 

14.88 

Inferential  Statistics 


None  of  the  covariates  accounted  for  a  significant  proportion 
of  the  within  cell  variance;  Wilks  Lambda  =  .16,  approximate 
F(20,  27.5)=1.03,  p.>.05.  Unfortunately,  complete  covariate  data 
were  only  available  for  19  subjects;  EL  and  SC  scores  were 
missing  in  10  cases.  Therefore,  EL  and  SC  scores  were  dropped 
from  all  further  analyses. 

The  MANOVA  was  reexecuted  with  the  three  remaining  covariates. 
The  covariates  did  account  for  a  significant  proportion  of  the 
within  cell  variance.  This  is  demonstrated  by  the  MANOVA 
statistics:  Wilks  Lambda=  .35,  approximate  F(12,50.6)=  2.07, 
p.<.05.  This  finding  is  explained  by  the  within  cell  regression 
analyses,  which  indicated  a  multiple  R  of  .56,  and  an  adjusted  R 
of  .22  between  nonfatal  errors  and  the  covariates,  F  (3, 22) =3. 28, 
p.<.05.  The  regression  analyses  between  the  other  dependent 
variables  and  the  covariates  were  not  significant. 

The  only  covariate  to  account  for  a  significant  proportion  of 
the  within  cell  variance  for  nonfatal  errors  was  the  TABE 
Vocabulary  scores,  t(3,22)=  2.54,  p.<.02.  However,  there  was  no 
main  effect  for  condition  on  the  nonfatal  error  variable.  Thus, 
the  covariates  were  not  considered  in  any  succeeding  analyses. 

The  MANOVA  indicated  that  the  independent  variable,  job  aid 
condition,  accounted  for  a  significant  proportion  of  the 
variance.  A  Wilks  Lambda  value  of  .22  was  obtained  with  an 
approximate  F(8,38)=5.33,  p.<.0001.  The  follow-up  univariate  F 
tests  indicated  that  the  groups  differed  significantly  across 
time,  F(2,22)=  23.90,  p.<.0001,  and  condensed  survey  responses, 
F(2,22)=  5.19,  p.<.02.  According  to  the  univariate  F  tests  the 
groups  did  not  differ  significantly  across  fatal  and  nonfatal 
errors. 

Post  hoc  tests  were  used  to  determine  the  nature  of  the  group 
differences  for  the  time  and  survey  variables.  The  SPSS  oneway 
procedure  was  used  to  identify  differences  between  the  three 
groups.  The  Sheffe  procedure  indicated  that  all  three  groups 
differed  from  each  other  at  the  p=.05  level,  see  Table  2.  Both 
job  aid  groups  produced  faster  performance  than  the  TM  +  Demo 
group.  Also,  the  Job  Aid  +  Demo  group  performed  faster  than  the 
Job  Aid  Only  group. 
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Table  2 


Post  Hoc  Comparison  of  Groups  across  the  Time  Variable 


Comparison 

F-obtained 

JA  Only, 

JA+Demo 

7.15* 

JA  Only, 

TM+Demo 

13.66** 

JA+Demo . 

TM+Demo 

41.81*** 

Note.  Critical  Value-Schef fe  (Ferguson, 1976) . 
*  p<.05  **  p<.01  ***  p<.001 


For  the  condensed  survey  variable,  both  job  aid  groups 
indicated  greater  satisfaction/confidence  than  the  TM  +  Demo 
group.  The  difference  between  the  Job  Aid  +  Demo  and  Job  Aid 
Only  groups  failed  to  reach  significance  according  to  the  Scheffe 
procedure.  See  Table  3. 

Table  3 

Post  Hoc  Comparison  of  Groups  across  the  Combined  Survey 
Question 

Comparison _ F-obtained 

JA  Only,  JA+Demo  .00 

JA  Only,  TM+Demo  11.43** 

JA+Demo.  TM+Demo _ 12 . 34** 

Note.  Critical  Value-Schef fe  (Ferguson,  1976) . 

**  p<.01 

The  two  demo  groups  were  compared  on  the  survey  question  that 
requested  subjects  to  rate  the  effectiveness  of  their  demo.  A 
significant  group  difference  was  not  demonstrated  for  this 
question,  F(l, 14)=1.99,  p>.05. 

Parameter  Estimates 


The  group  means  and  variances  were  used  to  estimate  the  size 
of  the  effect  of  the  Job  Aid  Only  group  and  the  Job  Aid  +  Demo 
group.  Due  to  unecjual  variances,  effect  sizes  were  reported  with 
respect  to  the  variance  of  the  TM  +  Demo  group  according  to  the 
formula:  Effect  Size=  (mean  experimental-mean  control)  /  sd 
control  (Bloom,  1984) .  Comparisons  between  the  JA  +  Demo  and  the 
Job  Aid  Only  group  were  made  with  respect  to  the  variance  of  the 
Job  Aid  Only  group.  Refer  to  Table  4  for  effect  size  estimates. 
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Table  4 


Effect  Size  Estimates 


Grouos  comoared 

Time 

Survev 

Nonfatal 

Fatal 

JA+Demo, 

TM+Demo 

2.44 

1.09 

.22 

.54 

JA  Only, 

TM+Demo 

1.38 

1.07 

.78 

.51 

JA+Demo . 

JA  Onlv 

1.23 

.03 

-.66 

.07 

For  the  time  variable,  large  effect  size  estimates  were 
demonstrated  for  the  Job  Aid  -h  Demo  and  Job  Aid  Only  groups 
relative  to  TM  +  Demo  group,  2.44,  1.38,  respectively.  A  large 
effect  size  was  also  demonstrated  between  the  Job  Aid  +  Demo  and 
the  Job  Aid  Only  group,  1.23. 

For  the  condensed  survey  question,  large  effect  size  estimates 
were  demonstrated  between  the  Job  Aid  +  Demo  and  Job  Aid  Only 
groups  relative  to  TM  +  Demo  group,  1.09  and  1.07.  A  small 
effect  size  was  observed  between  the  Job  Aid  +  Demo  and  the  Job 
Aid  Only  group,  .03. 

Although  significant  differences  were  not  demonstrated,  effect 
size  estimates  were  calculated  for  the  fatal  and  nonfatal  error 
variables.  Medium  to  small  effect  sizes  were  found  for  Job  Aid+ 
Demo  and  Job  Aid  Only  groups  relative  to  TM  +  Demo  group  for  the 
nonfatal  error  variable,  .22,  and  .78,  respectively.  Medium 
effect  sizes  were  also  found  for  Job  Aid  +  Demo  and  Job  Aid  Only 
groups  relative  to  TM  +  Demo  group  for  fatal  errors  .54,  and  .51, 
respectively. 

Because  significant  effects  were  not  demonstrated  for  fatal 
and  nonfatal  errors,  power  analyses  were  performed  to  estimate 
the  power  of  the  research  design  given  the  mean  differences  and 
the  variances  that  were  obtained  for  these  variables.  Effect 
size  was  estimated  via  a  formula  for  three  or  more  groups  given 
in  Cohen  (1977) ; 

Effect  Size=  sd  -  mean/sd  within  cell. 

Effect  sizes  of  .33  and  .40  were  estimated  for  the  fatal  and 
nonfatal  error  variables,  respectively,  on  the  basis  of  the  mean 
differences  of  the  three  groups  .  Given  sample  sizes  of  ten,  the 
probability  of  obtaining  significant  effects  at  the  .05  level 
(one-tail)  was  estimated  to  be  31  and  45  percent  for  the  two 
means. 
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Discussion 


Effect  of  the  Job  Aid 


The  first  major  hypothesis  was  that  soldiers  using  the  ARI  job 
aid  would  display  faster  and  more  accurate  performance  than 
soldiers  using  the  Army  TM.  This  expectation  is  confirmed  by  the 
finding  that  both  job  aid  groups  performed  the  MSRT  procedures 
faster  than  the  TM  +  Demo  group.  Effect  size  estimates  are 
consistent  with  the  view  that  using  the  Job  Aid  resulted  in  a 
large  effect  on  the  time  reofuired  to  complete  the  MSRT 
procedures.  A  two  standard  deviation  group  difference  was 
observed  between  the  TM  +  Demo  and  the  Job  Aid  +  Demo  groups. 

This  indicates  that  the  average  subject  in  the  Job  Aid  +  Demo 
group  performed  at  a  level  equivalent  to  the  ninety-eighth 
percentile  of  the  TM  +  Demo  group. 

The  survey  data  are  consistent  with  the  time  data  and  indicate 
that  the  use  of  the  Job  Aid  led  to  greater  satisfaction/ 
confidence  with  the  MSRT  task  than  use  of  the  TM.  The  two  job 
aid  groups  rated  the  task  as  easier,  and  claimed  more  confidence 
in  their  future  ability  to  perform  the  procedures  than  did  the  TM 
group.  Also,  the  job  aid  was  rated  more  effective  than  the  TM. 
Large  effect  sizes  were  observed  between  the  both  job  aid 
conditions  and  the  TM  +  Demo  group  on  the  survey  questions. 

It  was  also  expected  that  use  of  the  job  aid  would  lead  to 
fewer  errors  in  performing  MSRT  procedures  than  use  of  the  TM. 
Mean  differences  between  the  three  groups  are  consistent  with 
this  expectation,  although  the  sample  size  was  not  large  enough 
to  demonstrate  statistical  significance.  Moderate  effect  sizes 
were  demonstrated  between  the  Job  Aid  +  Demo  and  TM  +  Demo  groups 
on  both  error  variables.  The  power  analysis  indicates  that 
effect  sizes  in  this  range  have  a  low  probability  of  being 
successfully  demonstrated  with  a  sample  size  of  29.  It  should  be 
noted  that  the  ratio  of  the  difference  between  perfect 
performance  and  the  performance  of  the  TM  +  Demo  group  to  the 
standard  deviation  of  the  TM  +  Demo  group  is  equal  to  1.18 
(5.90/5.00)  and  1.52  (4.10/2.69)  for  fatal  and  nonfatal  error 
variables.  Given  that  the  job  aid  can  not  be  expected  to  lead  to 
perfect  performance,  it  is  not  surprising  that  the  data  failed  to 
indicate  a  statistically  significant  improvement  in  error  rates 
due  to  the  job  aid.  This  pattern  of  results  across  four 
dependent  variables  indicates  that  the  job  aid  had  a  large 
positive  effect  on  MSRT  operations. 

Effect  of  the  Demonstration 


The  second  major  hypothesis,  that  the  demonstration  would 
result  in  an  improvement  in  performance  was  also  supported  by  the 
inferential  statistics  collected  for  the  time  data.  Of  the 
soldiers  who  used  the  job  aid,  those  who  received  a  demonstration 
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soldiers  who  used  the  job  aid,  those  who  received  a  demonstration 
performed  faster  than  those  who  did  not.  It  should  be  mentioned 
that  the  Job  Aid  Only  group  still  performed  faster  than  the  TM  + 
Demo  group,  so  it  can  be  predicted  that  even  when  a  demonstration 
is  not  possible  the  Job  Aid  is  preferable  as  a  performance  aid  to 
the  TM.  The  fact  that  the  demonstration  did  not  result  in  a 
significant  effect  in  terms  of  the  number  of  nonfatal  and  fatal 
errors,  is  explained  by  the  power  analysis. 

Future  research  recommendations  include  use  of  larger  samples 
and  different  subject  types.  These  data  are  consistent  with  the 
expectation  that  larger  sample  sizes  would  have  revealed  effects 
for  the  fatal  and  nonfatal  error  variables.  In  the  future,  a 
similar  evaluation  using  Signal  MOS  soldiers  would  be  of  some 
benefit,  because  of  course  they  will  be  charged  with  operation  of 
the  MSRT.  Also,  the  User's  Manual  for  Predicting  Military  Task 
Retention  (Rose,  et  al.,  1985)  suggests  that  job  aids  are 
beneficial  when  performance  of  the  task  is  infrequent,  such  as  is 
usually  the  case  for  the  non-Signal  soldier  operating  the  MSRT. 
Although  the  data  show  that  the  job  aid  benefits  an  untrained 
soldier,  a  study  of  the  job  aid's  effect  on  skill  decay  would 
provide  additional  information. 

The  Job  Aid  developed  by  ARI  with  assistance  from  SLD,  both 
with  and  without  a  demonstration,  has  proven  effective  in 
diminishing  the  time  needed  to  complete  MSRT  operating 
procedures,  when  compared  with  the  traditional  Army  TM.  It  has 
also  been  shown  that  soldiers  generally  are  more  satisfied  with, 
and  confident  about,  their  performance  on  the  MSRT  when  using  the 
Job  Aid,  than  when  using  the  TM.  In  addition,  the  short 
demonstration  has  been  shown  to  significantly  augment  the  effect 
of  the  Job  Aid.  These  findings  indicate  that  the  Job  Aid, 
especially  when  accompanied  by  a  short  demonstration,  would  be  a 
beneficial  addition  to  the  training  and  supplementation  regimen 
of  the  non-signal  MOS  user  of  the  MSRT. 
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APPENDIX  A 


MSRT  JOB  PERFORMANCE  AID 
INSTRUCTIONS 


1.  This  Job  Aid  covers  six  common  operating  procedures  for  the  Mobile  Subscriber  Radio-Telephone  Terminal 
(MSRT).  The  name  of  each  procedure  is  presented  in  all  capitals  and  is  underlined. 

Z.  Page  2  contains  the  INDEX.  Before  performing  a  procedure  you  should  check  the  INDEX  for  its  page 
ncmber  and  any  other  important  information. 

3.  Page  3  contains  pictorial  representations  of  the  components  of  the  MSRT.  Refer  to  this  page  when  you 
are  uncertain  which  component  is  the  "RADIO",  etc.,  and  to  locate  dials  and  switches  on  the  components. 

4.  When  you  locate  the  Procedure  that  you  wish  to  perform,  you  will  see  that  it  is  divided  into  three 
coluins:  ACTION,  INDICATIONS,  and  CORRECTIVE  ACTION. 

For  each  tTunbered  ACTION  (1,2,3...)  or  subACTION  (a,b,c...): 

a.  Read  the  ACTION  coluin  first. 

b.  Then  read  the  INDICATIONS  column  that  corresponds  to  it. 

c.  Next,  perform  the  ACTION. 

d.  Check  for  the  INDICATIONS.  If  you  observed  the  proper 
INDICATION,  then  you  are  ready  to  read  and  perform  the 
next  numbered  ACTION. 

e.  Whenever  the  proper  INDICATION  is  not  observed,  or  an 
improper  indication  is  observed,  the  CORRECTIVE  ACTION 
must  be  taken.  When  CORRECTIVE  ACTION  is  taken,  proper 
INDICATION  should  appear  and  you  can  go  on  to  the  next 
ACTION. 

f.  If  you  run  into  a  problem,  read  the  CORRECTIVE  ACTIONS 
carefully  and  perform  them  thoroughly.  Some  CORRECTIVE 
ACTIONS  require  that  you  repeat  several  ACTIONS  in 
sequence. 

g.  Some  ACTIONS  do  not  have  INDICATIONS  corresponding  to 
them.  Simply  go  on  to  the  next  ACTION. 


A-1 


MSRT  JOB  PERFORMANCE  AID 


INDEX 


COMPONENTS  OF  THE  MSRT . P3 

PROCEDURE  1:  POWER  UP . P6 


*NOTE:  Steps  2  &  3  apply  to  MSRTs  not  previously 
loaded  with  crypto  keys,  or  when  changing  crypto  keys. 
Perform  these  steps  only  when  the  CRYPTO  ALARM  indicator 
is  ON,  or  when  you  have  been  instructed  to  change  a  key. 


PROCEDURE  2:  LOAD  CRYPTO  M  KEY  IN  RADIO . PA 

PROCEDURE  3:  LOAD  CRYPTO  KEYS  IN  TELEPHONE . P5 


*NOTE:  Step  4  applies  to  MSRTs  that  have  not  been 
loaded  with  frequency  plans.  Perform  this  step  only 
if  LOADED  FREQUENCY  PLAN  indicator  is  OFF. 

PROCEDURE  4:  LOAD  FREQUENCY  PLAN  (MANUAL) . P7 

*NOTE:  Step  5  applies  to  MSRTs  that  are  not 
affiliated  with  the  network.  Perform  this  step 
only  if  the  MARKER/AFFILIATED  indicator  is  OFF. 


PROCEDURE  5:  AFFILIATION . P7 

PROCEDURE  6:  HAKE  A  CALL . P8 


RT '!  539  RADIO 


DSVT  TELEPHONE 


ACTION 


CORRECTIVE  ACTION 


INDICATIONS 


POWER  UP 

1.  Ensure  FREQ  PLAN 
dial  on  RADIO  is  in 
NETWORK  position. 


2.  Ensure  RAU/MSRT/ 
Remote  Control  Dial 
is  in  upright  (12 
O'clock)  position. 

3.  Turn  POWER  switch 
to  ON  (located  above 
TELEPHONE  in  vehicle, 
or  below  RADIO  in 
standalone) . 

4.  Turn  RADIO  Dial 
labeled  OFF/BLACKOUT/ 
ON  to  ON. 


LOAD  CRYPTO  M 
KEY  IN  RADIO 

1.  Ensure  FILL/ZERO 
Switch  on  RADIO  is  in 
center  position  (To 
change  position,  pull 
switch  out  to  move). 

2.  Turn  KYK13  Z/ON/OFF 
Dial  to  OFF. 

3.  Set  KYK13  Z/ALL/ 
654321  Dial  to  number  of 
M  key  (ntinber  of  M  key 
is  written  on  side  of 
KYK13). 

4.  Push  gray  button 
KYK13  to  check  that 
key  is  inside. 


1.  Dial  is  in  NETWORK 
pos i t i on . 


2.  Dial  is  in  upright 
pos i t i on . 


3.  Switch  stays  on. 


a.  If  cryptovariables 
have  not  been  loaded, 
indicators  are  off, 
except  for  the 
RADIO'S  CRYPTO 
ALARM,  which  is  on. 

b.  If  cryptovariables 
have  been  loaded,  all 
indicators  are  off. 

C.  If  CRYPTO  ALARM 
is  flashing,  take 
corrective  action. 


4.  Parity  indicator 
above  gray  button  will 
flash  (red  light). 


1 .  If  not  in  position, 

DO  NOT  ATTEMPT  TO 
CHANGE  POSITION.  Call 
BSO/Maintenance. 

2.  If  not  in  an  upright 
position,  change  position 
with  flathead  screwdriver. 


3.  If  switch  trips  off, 
call  BSO/Maintenance. 


a.  Zero  RADIO  by  moving 
ZERO/FILL  Switch  to 
ZERO,  then  return  to  center 
position. 


b.  Same  corrective  action  as 
4a  above. 


c.  If  cables  are  connected 
call  BSO/Maintenance. 


4.  If  parity  indicator 
does  not  flash,  notify 
supervisor  that  KYK13 
is  not  loaded. 
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ACTION 


INDICATIONS 


CORRECTION  ACTION 


5.  Connect  KYK13  to 
J3  f i I '  connector  on 
RADIO  (push  KYK13  in 
and  to  the  right). 

6.  Turn  KYK13  Z/ON/ 
OFF  Dial  to  ON. 

7.  To  load  M  Key: 
indicator  goes  off. 

Push  FILL/ZERO 
Switch  on  RADIO  up 
4  times. 

8.  Turn  KYK13  Z/ 
ON/OFF  Dial  to  OFF. 

9.  Remove  KYK13  from 
fill  connector. 

LOAD  CRYPTO  KEYS  IN 
TELEPHONE 

1.  Ensure  VAR  STOR 
Switch  on  TELEPHONE 
is  in  center  position 
(to  change  position, 
pul  I  up  to  move) . 

2.  To  Load  U  Key: 

a.  Turn  KYK13 
Z/ON/OFF  Dial  to  OFF. 

b.  Set  KYK13 
Z/ALL/654321  Dial 
to  number  of  U  key. 


c.  Push  gray  button 
on  KYK13  to  check  that 
key  is  inside. 


7.  CRYPTO  ALARM 
call  BSO/Ma i ntenance . 


b.  Nutiber  of  U  and  M 
keys  are  written  on 
back  of  KYK13,  or  ask 
supervisor. 

c.  Parity  indicator 
above  gray  button  wi 1 1 
flash  (red  light). 


d.  Connect  KYK13  to 
fill  connector  on 
TELEPHONE  (KYK13  upside 
down,  but  facing 

back  seat). 

e.  Turn  KYK13 
Z/CN/OFF  CHECK  Dial 
to  ON. 


7.  If  indicator  stays  on. 


c.  If  parity  indicator 
does  not  flash,  notify 
supervisor  that  KYK13 
is  not  loaded. 


ACTION 

f.  Set  FUNCTION 
SEL  switch  on 
TELEPHONE  to  LOU. 


g.  VAR  STOP 

Switch  on  TELEPHONE  to 
LOAD  and  release  it. 


3.  To  Load  H  Key: 

a.  Set  KYK13 
Z/ALL/654321  Dial  to 
nunber  of  M  key. 

b.  Set  FUNCTION 
SEL  Switch  on 
TELEPHONE  TO  LOX. 

c.  Move  VAR  STOP 
Switch  on  TELEPHONE 
to  LOAD  and  release 
it. 


d.  Move  FUNCTION 
SEL  Switch  on 
TELEPHONE  to  OP. 


e.  Turn  KYK13 
Z/ON/OFF  Dial  to  OFF. 

f.  Remove  KYIC13  from 
fill  connector. 


INDICATIONS 

f.  RING/BUSY 
indicator  is  on 
and  NSW  is  flashing. 


g.  Two  tons  wi 1 1  be 
heard;  one  at  the 
LOAD  position  arxJ 
one  at  the 
center  position. 


b.  RING/BUSY  and  NSW 
indicators  on  TELEPHONE 
are  both  on. 

c.  Two  tones  will  be 
heard;  one  at  the 
one  at  the  center 
position. 


d.  RING/BUSY  indi¬ 
cator  is  on  and  NSW 
indicator  is  flash¬ 
ing. 


CORRECTIVE  ACTION 

f.  If  indicators  are  off, 
move  FUNCTION  SEL 
diaUon  TELEPHONE)  to  DSBL, 
move  VAR  STOR  Switch  to  ZERO, 
then  back  to  center.  Repeat 
action  2f,-  if  problem 

still  exists,  call 
BSO/Maintenance. 

g.  If  two  tones  are  not 
heard,  take  some  correc¬ 
tive  action  as  2f  above. 


c.  If  two  tones  are  not 
heard.  Zero  DSVT  by  moving 
FUNCTION  SEL  Switch  to 
DSBL.  Move  VAR  STOR 
Switch  to  ZERO  and  then 
back  to  center.  Repeat 
Action  3. 

d.  If  both  indicators  are 
off,  take  corrective 
action  3c  above. 
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ACTION 


INDICATION 


CORRECTIVE  ACTION 


LOAD  FREQUENCY  PLAN 
(MANUAL) 

1.  Lift  handset  of 
TELEPHONE. 

2.  If  misdial  occurs, 
hang  up  and  redial . 

Freq  Plan  No.  <2  digit  no.)  + 
all  Freq.  Plan  Pairs  in  sequence  + 

R 


2.  On  TELEPHONE,  enter: 
8F  ♦ 


3.  Hang  Up 

4.  Inspect  LOADED 
FREQ  PLAN  indicator 
on  RADIO. 


AFFILIATION 

1.  Lift  handset 
on  TELEPHONE. 


2.  Enter  on  TELEPHONE: 
8R  ♦ 

Personal  Code  + 

7  digit  number 
of  HSRT  you  are 
operating 

3.  Listen  for  error 
tone. 


4.  HANG  UP 
IMMEDIATELY 


a.  LOADED  FREQ 

PLAN  indicator  is  on. 

If  indicator  is  flashing, 
corrective  action. 

b.  LOADED  FREQ 

PLAN  indicator  is  off. 


1.  Error  tone  is  heard. 


3.  Tone  is  heard  in 
handset  as  each  digit 
is  entered.  Error  tone 
is  heard  when  dialing 
is  completed. 

a.  LOADED  DIR  NO. 
indicator  on  RADIO 
is  on. 


a.  Repeat  actions  1  &  2. 


b.  Notify  supervisor,  or 
BSO/Ma i nt enance . 


1.  Check  phone  connections. 

If  wires  are  connected  correctly, 
check  hook  switch.  Hake  sure  it  is 
unlocked--if  so,  call 
BSO/Maintenance. 


3.  Go  on,  then  off  hook. 
Reenter  8R  ♦  PC  +  7  digit 
nunber.  If  the  problem 
still  exists  cal  I  BSO/ 
Maintenance. 

a.  Same  as  3  above 
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ACTION 


MAKE  A  CALL 

1.  Lift  handset  on 
TELEPHONE. 


2.  On  TELEPHONE, 
enter  7  digit  nuit)er 
of  desired  party. 


INDICATION 

b.  TRAFFIC/SCANN 
indicator  on  RADIO 
F I ashes - - search i ng 
for  RAU  marker. 

c.  When  RAU  marker  is 
found,  TRAFFIC/SCANN 
indicator  is  on  solid. 

d.  When  Node  Center 
verifies  numbers, 
HARKER/AFFILIATION 
on  RADIO  is  on  solid. 


1.  Dial  tone  is  heard 
in  handset. 


a.  Call  is  completed. 
(Waiting  tone  is  heard 
while  network  searches 
for  nunber--make  take 
several  seconds). 

b.  Busy  tone  is  heard 
(Subscriber  called  is 
busy  or  network  is 
saturated). 


CORRECTIVE  ACTION 
b.  Same  as  3  above. 


1.  Check  status  of 
network.  Check  alarm 
indicator  on  RADIO; 
if  flashing  or  on, 
call  BSO/Maintenance. 

a.  If  dial  tone  is 
heard  while  trying  to 
place  call,  hang  up  and 
redial.  If  problem  still 
exists  call  BSO/ 
Maintenance. 
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Appendix  B 

Satisfaction/Confidence  questionnaire 


A.  Have  you  operated  or  affiliated  a  voice  secure  telephone 
before?  If  yes,  list  your  experience. 

1.  Rate  the  effectiveness  of  the  demo.  (For  2  demonstration 
groups  only) 

_  Exceptionally  good 

_  Reasonably  good 

_  So~So 

_  Reasonably  poor 

_  Exceptionally  poor 

Can  you  suggest  any  improvements  to  the  demo? 

2.  Rate  the  effectiveness  of  the  job  aid/TM.  (depending  upon 
group  assignment) 

_  Exceptionally  good 

_  Reasonably  good 

_  So-So 

_  Reasonably  poor 

_  Exceptionally  poor 

Can  you  suggest  any  improvements  to  the  job  aid/TM? 

3.  Rate  the  task. 

_  Very  easy 

_  Easy 

_  Neither  easy  nor  difficult 

_  Difficult 

_  Very  difficult 

What  was  the  most  difficult  part  of  the  task? 

4.  How  confident  are  you  that  you  could  successfully  affiliate 
and  complete  a  call  using  this  job  aid/TM? 

_  Very  confident 

_  Confident 

_  Neither  confident  nor  unsure 

_  Unsure 

_  Very  unsure 
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